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Abstract: 

In many digital image processing applications, image segmentation is require- 
initial partitioning of local image regions based on certain statistical or contex 
homogeneity measures. One goal of image segmentation would be to segmer 
into regions that correspond to physically and semantically coherent objects ii 
We propose an improved color segmentation algorithm by taking advantage c 
"k-mode" algorithm and an adaptive Bayesian k-means algorithm. The "k-mo- 
algorithm uses a physics-based distance metric to generate regular partitions 
color space. The adaptive k-means algorithm utilizes two additional mechanis 
spatial homogeneity constraints and spatial adaptivity, to achieve more robus 
coherent segmentation. The proposed algorithm integrates a physically more 
color space and the corresponding color difference metric into the the adaptiv 
K-means framework in an effort towards physics-based segmentation of phot 
color images 
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Towards Physics-based Segmentation of Photographic Color Images 



Jiebo Luo Robert T. Gray H$ien-Che Lee 
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Rochester, NY 14650, U.S.A. 



Abstract 

In many digital image processing applications, image seg- 
mentation is required to provide initial partitioning of local 
image regions based on certain statistical or contextual ho- 
mogeneity measures. One goal of image segmentation would 
be to segment the image into regions that correspond to phys- 
ically and semantically coherent objects in the scene. We 
propose an improved color segmentation algorithm by tak- 
ing advantage of a simple "k-mode" algorithm and an adap- 
tive Bayesian k-means algorithm. The "h-mode" algorithm 
uses a physics-based distance metric to generate regular par- 
titioning of the color space. The adaptive k-means algorithm 
utilizes two additional mechanisms, i.e., spatial homogene- 
ity constraints and spatial adaptivity, to achieve more ro- 
bust and coherent segmentation. The proposed algorithm 
integrates a physically more meaningful color space and the 
corresponding color difference metric into the the adaptive 
Bayesian k-means framework in an effort towards physics- 
based segmentation of photographic color images. 

1 Motivation 

Image segmentation is a frequently encountered problem 
in many image processing tasks. While a "perfect" segmen- 
tation is very difficult, if not impossible, to accomplish with- 
out certain higher level interpretation, there is no doubt that 
the intelligence of many digital image processing algorithms 
can be greatly improved upon the extraction of semantically 
meaningful regions from the image scenes. For example, seg- 
mentation of distinctive objects in consumer photographic 
images can lead to the identification of main subject(s) and 
the background, and thus the optimal determination of ex- 
posure compensation, Segmentation of human flesh is of 
particular importance to many applications involving the 
detection of people, such as automatic personnel verifica- 
tion, human-computer interaction, etc. 

Segmentation of color images has received significant at- 
tention from researchers [1, 2, 3, 4] because of the large 
amount of information contained in color images, compli- 
cated by such physical factors as surface orientation and 
illumination change, which make color image segmentation 
a very challenging task. In the past, many color segmenta- 
tion methods have assumed that color is a constant property 
of an object and color variation is due to random camera 



noise. Consequently, these methods segment images not 
only along the material boundaries but also along other 
boundaries exhibiting color variations, such as highlight and 
shadow boundaries, or object edges with significant shading 
chauges. Recently, physics-based segmentation has been at- 
tempted with initial successes [5, 6]. 

The proposed algorithm attempts to account for the 
physics of color variation in low-level statistical estimation 
rather than resorting to some higher level interpretation. 
This is achieved by the following: (1) selection of an ap- 
propriate color space, (2) design of a physics-based color 
difference measure, (3) coherent partitioning of the feature 
space, and (4) accommodation to local variation of the ob- 
ject properties and varying image capturing conditions. In 
short, the goal is to develop a general segmentation engine 
with good performance in terms of robustness to color vari- 
ations and noise, and reasonable computational complexity. 
Important features of the proposed algorithm include: 

• Utilization of a physics-based color space and color dif- 
ference metric 

• Utilization of spatial homogeneity constraints and spa- 
tial adaptivity 

• Synergy of the above for improved segmentation 

• Improved convergence 

2 Color space and color difference 
metric for segmentation 

The "k-mode" algorithm [5] is based on the partitioning 
of a carefully selected color space and a specific color differ- 
ence measure. This color difference measure helps achieve 
physically coherent partitioning of the selected color space. 
The color space partitioning resembles a Voronoi partition- 
ing with cells centered at the modes (or peaks) in the color 
histogram domain - 

It has been recognized that a proper color space is of fun- 
damental importance to color image segmentation [5, 7]. A 
color space transformation is equivalent to the feature ex- 
traction and feature selection processes typically involved in 
a pattern recognition task. In this study, a major concern is 
the relationship between the physical processes in the scene 
and the captured color image signals. It is desired that the 
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natural causes of color variation within a coherent object 
should be discounted while the variations of color between 
distinct objects should be retained. After theoretical anal- 
ysis and experimental tests, the (L, s t t) space was shown 
to be favored in most cases among a few eligible candidates 
[5]. Color spaces such as the CIE-LAB and CIE-LUV gen- 
erally yield perceptually consistent results, but they are not 
chosen because perceptual difference is not necessarily a bet- 
ter metric for separating physically different materials. In 
particular, significant perceptual differences can occur even 
within a physically coherent object due to the shading gen- 
erated by different lighting and surface orientation [7]. The 
(L t 5, t) space is given in Equation (1). It consists of one lu- 
minance component L t and two chrominance components s 
and t. We use (R t G, B) to represent the film density or log 
exposure space. The two chrominance components are pre- 
sumably invariant to intensity change of the lighting source. 
The s component approximately represents the illumination 
variation (daylight to tungsten light), and the t component 
is referred to as the green-magenta axis. 

L = -±=:(R + G + B) 
v3 

- = ^<*-B> (1) 

t = -L(#-2G + £) 

Traditionally, some distance metric is used to associate the 
color difference or color similarity with mathematical quan- 
tities [8]. The Euclidean distance between two points in the 
selected color space is often used as a measure of the sim- 
ilarity of the respective colors for segmentation purposes. 
Note that here the selected color space is explicitly stated 
because the distance metric is closely related to the color 
space. However, Euclidean distance in the RGB space has 
not been found to be a consistent measure of color similarity 
[5, 7]. 

To partition the color space into volumes that prefer- 
ably correspond to physically different objects, a few rules 
of thumb are considered appropriate. In general, the par- 
tition should be coarser in the luminance axis than in the 
chrominance axes. Furthermore, the partition should be 
coarser in color saturation than in hue. The former cor- 
responds to discounting the effect of luminance variation 
and the latter reduces the effect of specular reflection. The 
formula for this physics-based distance [5] is given in Equa- 
tion (2) where {Wi t W 9t Wh) are weighting factors for dif- 
ferences in the luminance, saturation and hue components, 
respectively. In general, we choose those weights such that 

Wi « w 9 < w h . 

d = y/Wi * (Alum) 2 + W, * (Asat) 2 + W h * (A/itie) 2 (2) 

The color difference measure is defined with respect to the 
mode (peak) of each class, m, and a given point c, in (s,t) 
plane. It is decomposed into a pseudo "saturation" com- 
ponent, AP, and a pseudo "hue" component, AQ. Intu- 
itively, "saturation" AP is parallel to the radial line from 
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Figure 1: A physics-based distance measure. 



each center, and "hue" AQ is perpendicular to that line, as 
illustrated in Fig. 1. 

3 The proposed algorithm 

The proposed algorithm is based on color space partition- 
ing with the incorporation of a Gibbs random field (GRF) 
model to account for spatial homogeneity constraints, and 
an adaptive mechanism to allow the local mean of each class 
to vary in spatial domain. The color of each region is mod- 
eled as a slowly varying non-parametric function plus noise. 
Spatial homogeneity constraints are imposed by a Gibbs 
random field model. Local color variations are accounted for 
in an iterative procedure involving the determination of the 
local class mean over a sliding window whose size decreases 
progressively. As a result, an image will be segmented into 
regions of uniform, or more often, slowly varying colors. 
This spatial adaptivity in essence attempts to account for 
color variations within objects, in addition to the adopted 
color difference metric. The original Bayesian adaptive seg- 
mentation algorithm for monochrome images was proposed 
by Pappas [9]. Chang et al. extended the Pappas algorithm 
to multichannel color images [7]. In this study, we mod- 
ified Chang's algorithm to incorporate the preferred color 
space and color difference measure determined in the de- 
velopment of the "k-mode n algorithm [5], and also to im- 
prove the speed of execution. By incorporating the preferred 
color space and color difference measure into the adaptive 
k- means algorithm, not only does the "k-mode" algorithm 
serve as the initialization stage of the adaptive k- means al- 
gorithm, but the physics-based difference measure is also 
adopted throughout the entire iterative segmentation pro- 
cess. It is expected that more homogeneous, coherent, ro- 
bust segmentation can be achieved. 

The original adaptive Bayesian color segmentation [7] is 
formulated as a MAP estimation, as given in Equation (3). 
The estimated segmentation x is defined as the one that 
maximizes the posterior probability of the segmentation x 
given the observed data y, i.e., p(x \ y). Using Bayes rule, 
we have 

x = argmaxp(x | y) = argmaxp(y | x)p(x), (3) 
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where p(y \ x) denotes the class conditional probability, and 
p(x) denotes the a priori probability of the expected seg- 
mentation. A Gibbs random field is utilized to model and 
enforce spatial smoothness constraints as the image prior 
[7, 9]. The assumption that the conditional distribution of 
each color component of each class is modeled by an indepen- 
dent Gaussian PDF (probability density function) [7, 9] im- 
plies the use of Euclidean distance because the log-likelihood 
of a specific color belonging to a certain class will be propor- 
tional to its Euclidean distance to the mean color defined 
by the Gaussian PDF. 

The introduction of the physics-based difference measure 
defined in Fig. 1 actually changes the original paradigm of 
MAP estimation. The new formulation can be considered as 
a constrained adaptive k-raeans clustering. There are three 
constraints in this sophisticated k-means procedure, namely, 
the spatial homogeneity constraints enforced by the GRF, 
the spatial adapt ivity enforced by locally estimated means, 
and color similarity measure enforced by the physics-based 
difference measure. Such a constrained clustering algorithm 
is expected to reach a good synergism capable of generat- 
ing a psychophysically coherent, robust, and noise resistant 
segmentation. 

4 Experimental Results 

With the incorporation of (L,s,t) color space and the 
physics-based color difference measure into the Bayesian 
adaptive k-means clustering, we obtain the class map in Fig. 
2(g) starting from the initialization of Fig. 2(b). Notice 
that a more homogeneous segmentation is achieved. The 
class map of Fig. 2(g) and the associated local class means, 
are then used to generate the rendering in Fig. 2(h). It is 
the capability of allowing local means to vary in the spatial 
domain, and the binding of spatially adjacent pixels by the 
Gibbs random field and the physics-based color difference 
metric, that enables the cohesive segmentation of the flesh 
region, which has quite different surface orientations and is 
lit by quite different lighting. 

FVom the experimental results with a Kodak image 
database, we have the observation that the marriage of the 
Bayesian adaptive k-means approach with the (L, s, t) color 
space and the physics-based color difference measure is well 
justified and does result in significant improvement in seg- 
mentation performance. It can be seen that the proposed 
method generates the most coherent segmentation results, 
as shown in Fig. 2(g) and 3(c). In particular, the white 
strip on the brown box emerges in the final segmentation of 
Fig. 3(c) even if it is not well defined in the initial segmenta- 
tion of Fig. 3(b); all the flesh regions are classified into one 
class. The somewhat artificial breakup of the table can be 
fixed by a simple perceptual grouping procedure that merges 
neighboring regions with no strong boundaries in between. 
Furthermore, the computation efficiency is significantly im- 
proved in the current implementation. For example, it takes 
a total of 192 seconds on a Sun Sparc 20 workstation to ob- 
tain the segmentation in Fig. 2(f) using Chang's algorithm. 



On the other hand, it takes only 7 seconds for the proposed 
algorithm to get Fig. 2(g). The faster convergence is partly 
due to the elimination of unnecessary iterations after the 
termination criterion is met for the current cycle. In addi- 
tion, we believe that the selections of the color space and 
color difference measure appropriate to segmentation con- 
tribute to the improved performance in terms of physically 
more coherent segmentation and faster convergence (fewer 
cycles and fewer iterations per cycle), although a more rigor- 
ous explanation is desired. Intuitively, color features corre- 
sponding to object color appearance are better clustered in 
the selected color space, and the clustering process is more 
effective using the selected color difference measure. 

5 Conclusion and Future Work 

In the future, a multiresolution implementation of the 
multichannel Bayesian adaptive k-means algorithm along 
the lines suggested by Pappas [9] is planned in order to 
further improve its speed and performance. It is expected 
that progressive coarse-to-fine segmentation will deter un- 
desired class splitting and unnecessary class change during 
the iterations. 

References 

[1] R. M. Haralick and L. G. Shapiro, "Image segmenta- 
tion techniques," CVGIP: Graphical Models and Image 
Processing, vol. 29, pp. 100-132, 1985. 

[2] G. Healy, "Using color for geometry- insensitive segmen- 
tation," J. Opt. Soc. Am. } vol. 6, no. 6, pp. 920-937, 
1989. 

[3] G. J. Klinker, S. A. Shafer, and T. Kanade, "A physical 
approach to color image understanding," Int. J. Corn- 
put Vision, vol. 4, no. 1, pp. 7-38, 1990. 

[4] Y. W. Lim and S. U. Lee, "On the color image segmenta- 
tion algorithm based on the thresholding and the fuzzy 
c-means techniques " Pattern Recognita vol, 21, no. 9, 
pp. 935-952, 1990. 

[5] H.-C. Lee, "Color image quantization based on physics 
and psychophysical J. Soc. Photogr. Sci. Technol. Jpn., 
vol. 59, no. 1, pp. 212-225, 1996. 

[6] B. A. Maxwell and S. A. Shafer, "Physics-based segmen- 
tation: Moving beyond color," in Conf. Comput. Vision 
Pattern. Reeognit, 1996. 

[7] M. M. Chang, M. I. Sezan, and A. M. Tekalp, "Adap- 
tive Bayesian estimation of color images,' 1 J. Electron. 
Imaging, vol. 3, pp. 404-414, October 1994. 

[8] R. W. G. Hunt, Measuring Colour. Chichester, England: 
Ellis Horwood, 1987. 

[9] T. Pappas, "An adaptive clustering algorithm for im- 
age segmentation," IEEE Trans. Signal Process., vol. 40, 
pp. 901-914, 1992. 



60 



Figure 2: Comparison of the segmentation results in raster scan order: (a) original image, (b) class map by 
"k-mode" segmentation, (c) class map by k-mcans clustering in RGB space, (d) class map by Bayesian adaptive k- 
means clustering in RGB space, (e) class map by k-means in CIELAB-like space, (f ) class map by Bayesian adaptive 
k-means clustering in CIELAB-likc space, (g) class map by the proposed algorithm, (h) rendered segmentation by 
the proposed algorithm. 




Figure 3: Segmentation results in raster scan order: (a) original image, (b) class map by "k-mode" segmentation, 
(c) class map by the proposed algorithm, (d) class map by the proposed algorithm after connected component 

analysis. 
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